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(57) Abstract 

A programmed digital computer (10) includes a display 
unit (16), an input unit (22), a processor (14), and a storage 
(20) for storing a program (102) which is executed by the 
processor (14). The program (102) includes a preprocessor 
(112) configured to convert a hypertext document (154) into 
a format which facilitates annotation. The document (154) 
is a text file in HyperText Markup Language (HTML) which 
represents a World Wide Web (WWW) page (38) that was 
downloaded from the internet (26) and is displayed on the 
display unit (16). The program (102) further includes an 
annotator (114) which enables a user to annotate the page (38) 
by using the input unit (22) to modify the document (154). 
Annotation tools include a graffiti pen (56) for drawing lines 
on the page (38), a highlighter (58) for highlighting selected 
text, digital sticker images (60) for pasting on the page, and 
cyber notes (62) which are similar to digital stickers (60). 
Annotated web pages (38) can be transmitted as electronic 
mail messages, and can be printed without sections being cut 
off. 
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SYSTEM FOR CAPTURING, ANNOTATING AND TRANSMITTING IMAGES OF INTERNET 

WEB PAGES 

TECHNICAL FIELD 

The present invention generally relates to the art of electronic communications, and more 
specifically to a system for capturing, annotating and transmitting images of internet World Wide Web 
(WWW) pages. 
BACKGROUND ART 

As a facet of the exponential explosion of digital computer and electronic communication 
access a huge volume of information and also to communicate directly with each other. This network 
is known as the "internet", and consists of a large number of computers which have information stored 
therein and are interconnected by telephone, satellite and other data links. 

These computers function as "servers", and can be accessed from an external location by a user 
with a personal computer to enable the user to obtain information stored therein. Most users connect 
their personal computers to the internet through service providers, although others are equipped to access 
the internet through direct connections. 

The most widely known and used functionality of the internet is known as the "World Wide 
Web (WWW)". In this system, the server computers are designated by addresses known as Uniform 
Resource Locators (URLs) and present their information to remote users in the form of "web pages". 
These web pages are created as text document files in the HyperText Markup Language (HTML), and 
include embedded hypertext links to graphic images, other URLs, etc. 

A remote personal computer user accesses the WWW using a software program known as a 
"browser," several of which, such as Microsoft* Internet Explorer and Netscape* Navigator, are currently 
available to individuals at no cost. The user types the URL to be accessed into the browser, or clicks 
on a link or icon representing the URL using a mouse. The browser then communicates with the URL 
over the internet and downloads a WWW page which is displayed by the browser on the computer 
monitor screen. 

The data which is actually downloaded from the URL to the user includes an HTML document 
file and graphic images which are referenced or pointed to by hypertext link elements in the HTML file. 
The browser interprets the HTML file, and generates and displays an image of the WWW page on the 
screen including any graphic images which are inserted at the locations designated by the elements in 
the HTML file. The displayed page can also include hypertext links in the form of highlighted text, 
icons, buttons, etc. which can be clicked on using a mouse to download another web Page, image, etc. 
from the same or another URL. 

It is desirable for a user to be able to save a complete web page and/or send a copy to a friend 
or business associate. However, a user-friendly way to provide this capability has not heretofore existed 
in the art. 

Although browsers provide an option to save a web page under a filename, all that is saved is 
the HTML document file. Graphic image files which constitute parts of a displayed web page, for 
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example, are not saved with the document file. Although graphic images are temporarily stored in a 
cache directory, this directory is periodically purged. 

If the saved HTML document is loaded into the browser after the cache is purged, the images 
will be missing from the displayed page. In order to restore the images, it is necessary to download 
them again from the Server (the URL is the address). Another limitation of conventional browsers is 
that they do not have any capability to annotate web pages prior to saving them, or save selected 
portions of web pages. 

If a user wants to send a copy of a web page to another person with annotations, it is necessary 
to perform a procedure such as printing out the web page, annotating it manually, and faxing it to the 
recipient. This is very cumbersome, time consuming, and "low-tech", and manual annotations made by 
a typical user can be seriously lacking in artistic quality. 

For this reason, there is a pressing need in the art to provide a system which will enable an 
internet user to save, annotate, and transmit a received web page in an efficient and aesthetic manner. 
DISCLOSURE OF THE IN VENTION 

The present invention fills the need which has existed heretofore in the art by providing a 
system for effectively capturing, annotating and transmitting images of internet World Wide Web 
(WWW) pages. 

In accordance with the present invention, a programmed digital computer includes a display 
unit, an input unit, a processor, and a storage for storing a program which is executed by the processor. 
The program includes a preprocessor configured to convert a hypertext document into a predetermined 
format which facilitates annotation. The document is a text file in HyperText Markup Language 
(HTML) which represents a World Wide Web (WWW) page that was downloaded from the internet and 

is displayed on the display unit. 

The program further includes an annotator which enables a user to annotate the page by using 
the input unit to modify the document. Annotation tools mclude a graffiti pen for drawing lines on the 
page, a highlighter for highlighting selected text, digital sticker images for pasting on the page, and 
cyber notes which are similar to digital stickers except that they further include boxes in which a user 
can type a text message. The annotations constitute image files which are painted on the page at 
positions designated by a mouse or other input device. 

Selected sections of web pages can be blocked, copied, and pasted into a scrapbook file. 
Annotated web pages can be transmitted as electronic mail messages, and can be printed without sections 
being cut off. 

The program can be provided in or added onto a user terminal as a stand-alone application or 
as a module (e.g. plug-in) of a web browser or other application, or can alternatively be provided in a 
server and accessed from a user terminal over the internet or other network. 

These and other features and advantages of the present invention will be apparent to those 
skilled in the art from the following detailed description, taken together with the accompanying 
drawings, in which like reference numerals refer to like parts. 
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RRTEF DESCRIPTION O F DRAWINGS 

FIG. 1 is a block diagram of a programmed digital computer which executes a software program 

according to the present invention; 

FIG. 2 is a diagram illustrating a World Wide Web (WWW) page displayed on a monitor screen 
of the computer of FIG. 1 according to the invention; 

FIG. 3 is similar to FIG. 2, but further illustrates a pull-down rile menu; 

FIG. 4 is also similar to FIG. 2, but illustrates annotation of the WWW page using a graffiti 

pen; 

FIG. 5 similarly illustrates annotation by highlighting text; 

FIGs. 6 and 7 illustrate pasting a cyber note on the page; 

FIG. 8 illustrates blocking text for copying to a scrapbook file; 

FIG. 9 illustrates a scrapbook file including the text blocked in FIG. 8; 

FIG. 10 is a block diagram illustrating an overview of the software system running on the 

computer of FIG. 1; 

FIG. 11 is a block diagram illustrating the main component elements of a software program 
according to the invention which is part of the system of FIG. 10; 

FIG. 12 is a block diagram illustrating a main module of the present software program; 

FIG. 13 is a block diagram illustrating annotation files representing digital stickers and cyber 
notes which can be pasted on displayed WWW pages using the present program; 

FIG. 14 is a block diagram illustrating the components of a WWW page which are stored in 

accordance with the invention; 

FIG. 15 is a flowchart illustrating a system tray operation of the invention; 

FIG. 16 is a flowchart illustrating a preprocessing operation; 

FIG. 17 is a flowchart illustrating a graffiti pen annotation operation; 

FIG. 18 is a flowchart illustrating a text highlighting operation; 

FIG. 19 is a flowchart illustrating a cyber note pasting operation; 

FIG. 20 is a flowchart illustrating an electronic mail transmission operation; 

FIG. 21 is a flowchart illustrating a printing operation; and 

FIGs. 22 to 25 are block diagrams illustrating server-based embodiments of the present 
invention. 

rbrt MOr>F. FOR CARPVTNG OUT THE INVENTION 

STRUCTURAL OVERVIEW 
The present provides a system for effectively capturing, annotating and transmitting images of 
internet World Wide Web (WWW) pages. The system includes a method which is embodied by a 
software program using a digital computer. The program is supplied to users on a computer-readable 

medium such as a CD-ROM. 

FIG. 1 illustrates a computer system 10 for implementing the present invention. In this 
embodiment of the invention, the system 10 is a stand-alone configuration. The system 10 includes a 
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Random Access Memory (RAM) 12 in which a software program for implementing the functionality of 
the invention is stored, and a processor 14 for executing the program. The system 10 further includes 
a visual display unit or monitor 16 for providing a visual display of relevant information on a screen, 
a read-only memory (ROM) 18 for storing firmware, and an input-output (I/O) unit 11 for connection 

to a printer, modem, etc. 

The system 10 further includes a mass data storage 20 which can be any combination of suitable 
elements such as a fixed (hard) magnetic disk drive, a removable (floppy) disk drive, an optical (CD- 
ROM) drive, etc. The program which implements the invention may be stored in the storage 20, and 
modules of the program loaded into the RAM 12 for execution as required. 

User access to the system 10 is provided by an input unit or device 22 such as alphanumeric 
keyboard and/or a pointing device such as a mouse. The elements of the system 10 are interconnected 
by a bus 24. The system 10 can be implemented by a conventional personal computer. 

As viewed in FIG. 1, the system 10 is connected to the internet 26 via a modem 28 and 
telephone line 30. However, other means of connection to the internet 26 are within the scope of the 
invention. The system 10 functions as a user terminal. 

The computer software program which implements the functionality of the present invention 
is a commercial product of Insight Development Corp. of San Ramon, CA entitled "Hot Off The Web 
(HOTW)". The program is currently supplied to end users on an optical CD-ROM disk in compressed 
form. It is expanded and loaded onto a hard drive of a user's computer by running a setup program. 
However, it is within the scope of the invention to provide the HOTW program on a computer-readable 
medium other than an optical element, such as on one or more magnetic elements such as floppy disks, 
or stored in a ROM or other computer memory element. 

As presented above, the present invention enables a user to capture, annotate, print and transmit 
images of WWW web pages downloaded from the internet 26. Each web page includes a hypertext 
document file in the HTML language, and ancillary files such as graphic image files which are 
referenced hy links in the document. As will be described in detail below, the HTML document is 
interpreted by a web browser, and the web page defined by the HTML document is displayed on the 
monitor 16 together with the graphic images represented by the image files inserted in their proper 
locations. 

The functionality of the present invention will be described first with reference to FIGs. 2 to 
9. Then, the internal structure and operation of the software will be presented in detail. 

FUNCTIONAL DESCRIPTION 

FIG. 2 illustrates the display on the monitor 16 generated by the HOTW program. The display 
includes a menu bar 32, toolbar 34, taskbar 35, icon tray 36, and a web page 38. 

The menu bar 32 includes text names of pull-down menus which can be accessed using a 
keyboard or mouse. The toolbar 34 is optional and provides a fast way to perform the operations which 
are also accessible from the pull-down menus. As illustrated, the toolbar 34 includes the following 
icons: "Open Page" 40, "Save" 42, "Print" 44, "Print Preview" 46, "Cut" 48, "Copy" 50, "Paste" 52, 
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"Undo" 54, "Graffiti Pen" 56, "Highlighter" 58, "Digital Sticker" 60, "Cyber Note" 62, "Scrapbook" 64 
and "E-mail" 66. These elements are omitted from FIGs. 3 to 9 except as necessary for simplicity of 

illustration. a 
In its currently marketed form, HOTW is designed to be used in conjunction with the Microsoft 
Intemet Explorer 4.0 web browser. It is a Windows* 95/98/NT 4.0 application, and is launched either 
by selecting the program from the "Start" menu or by clicking on an HOTW icon 68, which is located 
in the icon tray 36 in the right-hand corner of the taskbar 35. 

TRAY ICON 

If Internet Explorer is running, the user can click on the HOTW tray icon 68 using a monse, 
and the web page currently displayed in the orowser will be loaded into HOTW. The user may then 
open and work on multiple web pages simultaneously. 

If Internet Explorer is not running, a sample starter web page (the page 38 illustrated in FIG. 
1) containing a brief introduction to HOTW is displayed. 

PULL-DOWN MENUS 
FIG. 3 illustrates one of the pull-down menus, in this case a "File" menu 70, which is accessed 
by mouse clicking on "File" in the menu bar 32 or pressing Alt-F on a keyboard. "Open" brings up a 
dialog box which allows the user to enter a web address (URL) and open a page directly from the 
Internet If Internet Explorer is running, the current page will be displayed. 

A user may click on the arrow to the right of the URL to display a list of recently viewed 
pages, or click "Browse" to open a page from the user's file system. HOTW will open HTML files 
only. 

••Save" and "Save As" save the contents of the HOTW page into a folder in order to preserve 
all the different ancillary files that make up the page. A HOTW page is opened by opening the folder, 
then clicking on the HTML file that shares the same name as the folder. 

"Import" allows a user to import PNG, GIF or JPEG graphic image files for use as Cyber Notes 
and Digital Stickers as will be described below. "Print" causes the page to be printed. "Print Setup- 
contains options the user can use for standard adjustments such as orientation, paper size and source. 

Although not explicitly illustrated, the "Edit" pull-down menu includes the following selections. 

"Undo" reverses a previous action. 

"Cut" removes a blocked section and saves it to the clipboard. 
"Copy" copies a blocked section to the clipboard. 
"Paste" places the clipboard content into a page. 
"Delete" removes the selected content from the displayed page. 
"Select All" selects all of the content, including the web content and annotations. 
"Clear All Annotations" deletes all the HOTW annotations the user has made on a page. 
The "View" selection on the menu bar 32 accesses a pull-down menu (not shown) which 
enables the user to change the view and set preferences. 

ANNOTATION TOOLS 
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The "Annotate" selection on the menu bar 32 enables a user to select one of four annotation 
tools, and also to edit the text in a cyber note. The annotation tools are "Graffiti Pen", "Highlighter", 
"Digital Sticker" and "Cyber Note", and can also be activated by clicking on the icons 56 to 62 
respectively. 

After a tool is selected, a floating dialog box or palette appears which allows the user to make 
specific choices regarding the use of the tool (the descriptions appear under each individual tool). The 
palette is closed by clicking on the X in the upper right-hand corner of the screen or by selecting a 
different tool. 

The "Scrapbook" selection on the menu bar 32 enables a user to create a new scrapbook and 
annotate and organize portions of multiple web pages, add, edit and format a scrapbook title, and send 
selected content to an open scrapbook or a new scrapbook. 

GRAFFITI PEN 

The "Graffiti Pen" tool is illustrated in FIG. 4, and is activated from the "Annotate" menu or 
by clicking on the icon 56. A "Graffiti Pen" palette 72 pops up which displays the current pen choice 
in terms of size and color, and allows the user to change the attributes of both. This palette 72 also 
displays a selection of 15 pen colors. 

After the user chooses a pen size and color, the cursor changes into a cross-hair 74. To draw 
a line such as indicated at 76 on the displayed web page, the left mouse button is held down and the 
cursor 74 is moved around the screen. The mouse button is released to complete the drawing. 

HIGHLIGHTER 

As illustrated in FIG. 5, selection of the "Highlighter" tool causes a "Highlighter" palette 78 to 
pop up which allows a user to select the color of the highlight marker. The color selected is displayed 

at the top of the palette. 

When the user selects "Highlighter", the cursor changes into an I-beam 80 when the cursor is 
over text. The user clicks the left mouse button and drags the cursor 80 over text to be highlighted. 
The highlight appears when the mouse button is released. To change the highlight color, the user selects 
a new color from the palette 78 and repeats the selection. 

To immediately undo a highlight, the user may select "Undo" from the "Edit" menu or click on 
the "Undo" icon 54. Or, the user may select "None" from the palette 78, then select the text highlight 
the user wishes to erase by left-clicking and dragging. When the mouse button is released, the highlight 
will disappear. 

CYBER NOTES 

The Cyber Notes annotation tool is illustrated in FIG. 6. This tool enables a user's thoughts 
and comments to be added to a web page as an annotation along with professionally created images and 
graphics. More specifically, a Cyber Note is a graphic image or sticker which can be pasted onto a web 
page. Each Cyber Note includes a central box area into which the user can type a message using a 
keyboard. 

Activating the tool causes a "Cyber Notes" dialog box or palette 82 to pop up. The "Cyber 
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Notes" palette 82 is organized according to theme. The palette 82 includes a text window 83 which 
displays the name of the currently selected theme, in the illustrated example "Pop Secret Agents", and 
thumbnail images 87 of the Cyber Notes in the theme. Clicking on an arrow icon 85 causes another 
drop-down dialog box (not shown) to pop up which displays the names of all of the available themes. 

The set of thumbnails in a theme is viewed by selecting a theme using the scroll bar 89. By 
default, the displayed theme is the last theme used. This may be changed by selecting "Preferences" 
from the "View" menu, and choosing the theme to be used as the default. 

The most frequently used Cyber Notes can be collected in the "Favorites" theme for easy 
reference. To remove a Cyber Note from "Favorites", the user selects the thumbnail in the "Favorites- 
theme and clicks on the "Remove from Favorites" button. The user will still be able to find that Cyber 

Note in its original theme. 

After selecting a Cyber Note 86 from the palette 82 using a mouse click, the cursor changes 
into a rectangular outline 84 of the selected note 86. To place the note, the user moves the outline 84 
to the desired location on the page, then clicks the left mouse button. The Cyber Note then appears in 
the web page as indicated at 88 in FIG. 7. The cursor 80 appears in the Cyber Note 88, allowing the 
user to enter a text message 90 using a keyboard. 

Messages may be edited by double-clicking on the Cyber Note 88. A "Cyber Note Editor- 
dialog box (not shown) appears, allowing the user to format a message by typing text into a box in the 
cyber note 88. 

A user can import a graphic image file from a source outside the HOTW program (e.g. clip art) 
and use it as a Cyber Note. This is done by selecting "Import" from the File menu, and then selecting 
the "Cyber Notes" menu item, the user chooses the theme to add the note to, then browses the file 
system for the PNG, GIF, or JPEG graphic image file the user wishes to add. The user selects the file 

and clicks the "OK" button. 

A dialog box (not shown) appears which allows the user to set the text margins. The image 
being imported will be displayed, along with an outline representing the text box area for the Cyber 
Note. To adjust the text area, the outline is dragged to the size the user wants to be able to enter text 
in. 

DIGITAL STICKERS 

Although not explicitly illustrated, the "Digital Stickers" annotation tool is similar to the "Cyber 
Notes" tool in that it enables a user to add graphics to a web page to get points and messages across. 
A Digital Sticker is a graphic image which differs from a Cyber Note in that it does not include a text 
box for entering a user's message. Digital Stickers are arranged according to themes and are selected 
and pasted onto a web page in a manner which is essentially similar to the Cyber Notes as described 
above. External graphic images can be imported for use as Digital stickers in the same manner as for 
Cyber Notes. 
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SCRAPBOOK 

The "Scrapbook" tool allows the user to easily organize information found on the web. For 
example, if the user is shopping for a new car, the user can select the information needed from each of 
the automobile web sites visited and place each item of interest into a single scrapbook. The annotation 
tools can also be used to make notes on the content after it is brought into the scrapbook. 

The user can select "New Scrapbook" from the "Scrapbook" menu or click on the "Scrapbook" 
icon 64. A dialog box (not shown) appears which prompts the user to name the new scrapbook and to 
select an option for listing the URLs and times of each web page in the user's scrapbook. 

To add content to a scrapbook, the user selects items from a web page, then clicks the right 
mouse button and selects the "Send To Scrapbook" menu item, or selects "Send to Scrapbook" from the 
"Scrapbook" menu. 

A "Send to Scrapbook" dialog box (not shown) appears, listing the user's open scrapbooks. 
Selecting an open scrapbook automatically adds the content to the end of that scrapbook. Clicking "New 
Scrapbook" brings up the "New Scrapbook" dialog box described above under "New Scrapbook." Once 
the user names the new scrapbook, selects its URL and time stamp option, and clicks OK, the user has 
the option of copying or sending the selected content of the new scrapbook. The content is 
automatically added to the new scrapbook. 

As illustrated in FIG. 8, a section 92 of web page content can be blocked for selection using 
a monse. This brings up a dialog box 94 which enables the user to send the blocked content to a 
scrapbook. FIG. 9 illustrates the section 92 stored in a scrapbook named "My Scrapbook". HOTW 
further provides the option of automatically adding a stamp 96 following the content 92 which indicates 
the origin of the content and the time and date it was added to the scrapbook. 

Saving a scrapbook is like saving any other HOTW page. All of the related files are saved into 
a folder. When anew scrapbook is created, it is automatically saved in the "Scrapbook" subfolder under 
the "Hot Off The Web" program folder. To open an existing scrapbook, the folder is first opened, the 
HTML file that has the same name of the folder is selected. 

E-MAIL 

An annotated page can be sent to a friend or business associate using the electronic mail or "E- 
mail" function. This can be accessed by selecting "Send Mail" from the "File" menu or clicking on the 
icon 66. 

This causes a message composition form (not shown) to pop up from the user's default E-mail 
program with the HOTW page automatically attached as a self-extracting CABINET ("CAB") (.exe) file. 
If the recipient of the E-mail has HOTW, the attachment will be opened as a HOTW file. If not, the 
attachment will be opened in the recipient's default web browser. This annotated page may exist as an 
attachment, or as an integral part imbedded in the e-mail message. 

DETAILED STRUCTURE 

The internal structure of the present invention will now be presented in detail. FIG. 10 
illustrates the various software components which in combination provide the functionality of the 
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invention. 

As described above with reference to FIG. 1, the computer system 10 is a programmed digital 
computer including a digital data storage which collectively includes the data storage 20, RAM 12 and 
ROM 18. The software for practicing the invention is stored in the storage, and includes an operating 
system 100, in the present case Microsoft Windows* 95 or 98 or NT 4.0, the Hot Off The Web (HOTW) 
program 102, one or more web pages or HTML documents and ancillary files 104, and a web browser 
106. 

In the currently marketed version of HOTW the browser 106 is Microsoft* Internet Explorer 
4.0 although, as will be described below, the invention is not limited to any particular web browser, and 
can alternatively be used without a web browser. 

The detailed configuration of the browser 106 is not the particular subject matter of the present 
invention, and will be described only to the extent that is necessary for an understanding of the present 
invention. Full details regarding the Internet Explorer web browser are available from Microsoft. 

The browser 106 generally includes a browser control 106a which is used to access URLs, 
display menus and dialog boxes for various options, and download and display selected web pages from 
the internet. The browser 106 further includes a parsing and rendering engine 106b which is controlled 
by the browser control 106a for interpreting a downloaded document in HTML format and rendering 
a corresponding web page on the monitor 16, including all graphic and other ancillary files. 

The engine 106b in Internet Explorer is a file entitled MSHTML.DLL, which is also used as 
a stand-alone module by the HOTW program 102. Further included in the browser 106 are a suite of 
multimedia controls and plug-ins which process ActiveX scripts, Java Applets, etc. under control of the 
browser control 106a and engine 106b. 

The software components illustrated in FIG. 10 are stored in the various elements of the storage. 
Generally, these components will be stored on a hard drive of the data storage 20, and various modules 
will be loaded into the RAM 12 for execution by the processor 14 as required. The manner in which 
the software is stored is not limited within the scope of the invention. For example, although the 
preferred manner of providing the HOTW program 102 to a user is to store it on a CD_ROM and install 
it onto the computer storage, it is possible to store part or all of it in the ROM 18 or in an auxiliary 

ROM (not shown) as firmware. 

The main components of the HOTW program 102 are illustrated in FIG. 11. The program 102 
includes a tray icon module 1 10 which enables the user to launch the HOTW program by clicking on 
the tray icon 68 illustrated in FIG. 1. A preprocessor 112 is provided to convert an HTML text 
document file which is downloaded from the internet or obtained from another source into a 
predetermined format which facilitates annotation. 

An annotator or main program 114 provides overall control of the other components of the 
HOTW program 102 and enables the user to annotate a web page after preprocessing by the preprocessor 
112 using the input unit or devices 22 of FIG. 1 . The preprocessor 1 12 and program 1 14 use the engine 
106b to get and set attributes in HTML files using Microsoft's Document Object Model (DOM) 
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language. Detailed documentation on the manner in which DOM can be used to perform these 
operations is available from Microsoft. 

The HOTW program 102 further includes an E-mail module 116 for compressing a web page 
into a self-extracting CAB ( exe) file and transmitting it to a recipient as an attachment to an E-mail 
message. A printing module 11 8 . enables a web page to be printed without any portions thereof being 
cut off. 

The HOTW program 102 further includes a collection of annotation files, more specifically sets 
of Digital Stickers and Cyber Notes, which are organized by theme as described above. The parsing and 
rendering engine 106b, which is actually part of the web browser 106, is also illustrated in FIG. 11 
because in an alternative embodiment of the invention as will be described below, the engine 106 can 
be provided as part of the HOTW program 102 itself and used as a stand-alone component. 

The major components of the main module 114 are illustrated in FIG. 12. A controller 130 
provides overall control and coordination between the component modules for annotation and other 
operations. 

A module 132 uses the browser control 106a to generate the toolbars and icons which were 
described above with reference to FIGs. 2 to 9. The particular manner in which the toolbars and icons 
are generated is not the particular subject matter of the invention and will not be described in detail. 
Documentation on the manner in which the browser control 106a can be used to perform these 
operations is available from Microsoft. 

A module 134 also uses the browser 106a to generate the dialog boxes and palettes for the four 
annotation tools as described above. A module 136 is used to insert Digital Stickers and Cyber Notes 
into web pages using Cascading Style Sheets (CSS) as will be described in detail below. A module 138 
operates in conjunction with the module 136 to enable a user to position a Digital Sticker or Cyber Note 
at a desired position on a web page. 

A module 140 enables a user to import an external file (e.g. clip art) for use as a Digital Sticker 
or Cyber Note. A module 142 is used to create and edit a scrapbook, and insert selected portions of web 
pages into a scrapbook. A file manager module 144 is used by the other modules to open folders and 
files, save and print web pages, and perform other file oriented operations as required. 

FIG. 13 illustrates the format for storing the annotation files (Digital Stickers and Cyber Notes) 
120. The Digital Stickers and Cyber Notes for each theme are stored in separate folders. As illustrated, 
the Digital Stickers for a certain theme include a Digital Sticker HTML file 150 which includes 
hypertext elements or links that reference or point to the graphic image files for the Digital Stickers. 

The Digital Sticker images themselves are preferably in the Portable Network Graphic (PNG) 
file format (but can be GIF, JPEG or other type of file format) and have a filename extension of ".png". 
As shown, Digital Sticker images 150a to 150n referenced by the HTML file 150 are designated as 
"Digital_Sticker_l.png" to "Digital_Sticker_n.png" respectively. An element in the HTML file 150 
referencing or pointing to the file 150a, for example, is preferably provided as an element of a table in 
the HTML file 150, and will have a configuration such as: 
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<tr> 

<td align^'center" height="47" width="123"ximg 

src=" A%20Digital_Sticker_l .png" width=" 115" 
HEIGHT="84" alt="Digital Sticker l"></td> 

</tr> 

Further illustrated in FIG. 13 are a Cyber Notes HTML file 152 and PNG image files 152a to 
152n which are arranged in the same manner as the files 150 and 150a to 15 On. 

FIG. 14 illustrates the arrangement of an HTML document file and ancillary files as indicated 
at 104 in FIG. 10 which were downloaded from the internet. This collection of files is stored in its own 
folder. As illustrated, the files for one web page include a main hypertext document 154 in HTML 
language format which includes a link element to an ancillary graphics image file "Image_l.png" 154a. 

The HTML file 154 is also linked to image files 154b, 154c and 154d which are annotation files 
created by the user. The file 154b is for a Cyber Note and is named "Cyber Note_l.png"; the file 154c 
is for a Digital Sticker and is named "Digital_Sticker_l.png"; the file 154d is for Graffiti Pen annotation 
and is named »Graffiti_l.png". Although one file of each type is shown in the drawing, it will be 
understood that a particular HTML file 154 can reference no ancillary or annotation files, or alternatively 
any number thereof. 

DETAILED OPERATION 
The detailed operation of the present invention will now be described with reference to 
flowcharts illustrated in FIGs. 15 to 21. 

TRAY ICON 

FIG. 15 illustrates the operation of the tray icon module 110 which is illustrated in FIG. 11. 
The HOTW program 102 is preferably launched by mouse clicking on the HOTW tray icon 68 
illustrated in FIG. 2. The first step 160 is to determine if an instance of the tray icon 68 (the main 
program 114) is running. If so, the module 110 gets a web page (URL) from the open window of the 
browser 106 in a step 162. If not, the module 110 opens with the default page 38 as shown in FIG. 2. 
The module 110 then starts the maim HOTW program 114. 

PREPROCESSING 

The operation of the preprocessor 112 is illustrated in FIG. 16. The preprocessor 112 is called 
by the main program 1 14 after it has received control from the tray icon module 1 10 in a step 170. The 
preprocessor 112 first determines in a step 172 if the URL or web page is a local file (created locally 
or previously stored). If so, it copies the file and any ancillary graphic images or other files to a 
temporary directory in a step 174. If not, it copies the file(s) from the browser's cache memory (if 
available) or downloads it from a URL server via the internet in a step 176. 

The preprocessor 112 then parses the HTML file which is stored in the temporary directory to 
determine if any ancillary files are present in a step 178. The preprocessor 112 then preprocesses the 
HTML using the parsing and rendering engine 106b in a step 180. 

More specifically, the preprocessor 112 uses the parsing and rendering engine 106b and 
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Document Object Model (DOM) language to determine if a hypertext link which references an ancillary 
file of a predetermined type is present in the HTML document. The most common type of file which 
is linked to an HTML file, and desired to be included in the preprocessed web page, is a graphics file 
in a GIF, PNG, JPEG. Other formats such as AVI, Shockwave, Quicktime, wav or au, for sound 
graphics and video, may also be used since the scope of the invention includes linking and/or displaying 

any suitable type of ancillary file. 

The preprocessor 112 uses DOM language instructions to search for an element in the HTML 

document referencing or pointing to, for example, a graphics file. An element pointing to a GIF file 

named "Image_l.gif will have a typical format: 

<img src="http://www.mysite.com/images/Image_l .gif *> 

If a link is present, a local copy of the ancillary file is stored in the temporary directory, and 
the element referencing the ancillary file is modified to reference or point to the local copy. 

In the present example, the element pointing to the file "Image_l.gif was downloaded from 
a directory or folder named "images" at an URL named "http://www.mysite.com". The element is 
modified to reference or point to the file "Image_l.gif, but in this case in a local directory named 
"temp" on a "c:" hard drive in the storage 20 of the present system 10. The modified element will have 
the form: 

<img src="/temp/http_www_mysite.com_images_Image_l .gif' > 

The image file "Image_l.gif" itself is renamed as 
■•http_www_mysite.com_images_Image_l.gif • by changing the slashes "/" and dots »." to underscores 

Alternatively, the "http://" prefix can be dropped, and the file renamed as 
"www_mysite.com_images_Image_l .gif ' . 

If the folder "temp" can be implied as current, the element can be simplified by omitting the 

folder name and have the format: 

<img src="http__www_mysite.com_images_Image_l.gif> or 
<img src="www_mysite.com_images_Image_l.gif >■ 

After preprocessing of the HTML document, the ancillary files are renamed by the file manager 
144 to correspond to their new names in the HTML document and stored in the temporary directory or 
folder. If the user wants to save the web page with or without annotation, a new folder will be created 
with a name selected by the user, and the HTML file and ancillary files will be saved into the new 
folder. 

As presented above, the preprocessor 112 also modifies the HTML document to facilitate 
annotation by the controller 130 and associated modules of the main program 114. This modification 

includes a number of changes. 

1 The HTML content is placed into a fixed-width table which allows the annotation tool 

images to be easily inserted into the table. This also allows WYSIWYG printing preprocessed 
documents, which is preferably employed with the present invention. As such, the preprocessed HTML 
pages will have exactly the same layout regardless of the settings on the computers to which an HOTW 
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page is sent by E-mail and will also look like the actual annotations made. 

Since browser settings such as default font and window size affect the layout of an HTML 
document, the same HTML document may look quite different on different machines therefore 
compromising the effectiveness of annotations. The appearance of the displayed web page is fixed by 
defining the width of the encapsulating table. 

A fixed width table is created by inserting an element into the body of the HTML document 

having a format such as: 

<TABLE style="CURSOR: text" width-600> 

This element defines a table having a width of 600 pixels. The elements of the HTML 
document which generate the displayed web page follow the table definition element This allows the 
printing of a web page on an 8 1/2" x 11" sheet. 

2. Attributes designating fixed font properties (font and size) are assigned to all text 
elements. This causes the displayed page to have a predetermined appearance regardless of the default 
settings on the computer which is running the HOTW program, as described above. 
An element which performs this function has a form such as: 
<STYLE> HI { font-size: 24pt; } H2 { font-size: 18pt; } 
H3 { font-size: 13.5pt; } H4 { font-size: 12pt; } 
H5 { font-size: lOpt; } H6 { font-size: 7.5pt; } 
TABLE, TH, TR, TD { font-size: 12pt; } 
TT { font-size: lOpt; } 
SELECT { font-size: 9pt; } </STYLE> 

3. Java script and ActiveX elements are replaced by boxes of the same size as the original 
objects. Comments are placed in the boxes indicating their original functions. 

4. Marquee objects are made static. 

5. Buttons, forms, and mouse events are removed or disabled. 

ANNOTATION 

After the HTML and ancillary files have been preprocessed and stored in the temporary 
directory, the web page is ready for annotation by the user. The web page defined by the HTML 
document is rendered or displayed on the monitor 16 by the engine 16b together with the ancillary 
images in their proper locations. 

GRAFFITI PEN 

FIG. 17 illustrates the steps for annotating a web page using the Graffiti Pen. As described 
above, the user selects the Graffiti Pen tool from the menu bar or by clicking on the icon 56, and selects 
the color for lines to be drawn using the palette 72. These operations arc performed by the browser 
control 106a under control of the module 134 of the main program 114. 

To draw a line, the user holds down the left mouse button and drags the cursor over the 
displayed web page. The program 114 gets the mouse positions or points from the browser 106 in a 
step 190, and converts the points into pieces in a step 192. 
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Releasing the left mouse button provides an input to the program 1 14 which indicates that the 
line drawing operation has been completed. In response, the program 114 converts the pieces into a 
PNG file in a step 194 and stores the PNG file in the temporary directory as a position file together with 
the HTML and ancillary files in a step 196. 

The final step 198 is to insert an element into the HTML document which references the stored 
PNG file. This is preferably performed by the module 136 which inserts a Cascading Style Sheet (CSS) 
element into the page table of the HTML document with a position for the line and a Z-Order for the 
file specified. A Graffiti Pen placement element for a PNG file named "Graffiti_l.png" will have a 
format such as: 

<P style="CURSOR: auto"> <IMG border=0 src="Graffiti_l.png" 

style="CURSOR: auto; LEFT: 186px; POSITION: absolute; 
TOP: 114px; VISIBILITY: visible; Z-INDEX: 268"></P> 
The above is the format specified by Microsoft. However, it may not display properly in 
Netscape Navigator. A format which will work for Netscape is: 
<style type="text/css"> 

Graffiti_l.png { left: 186px; top: 114px; position: 
absolute; visibility: visible; z-index. 268; } 
</style> 
<body> 

<DIV ID="Graffiti_l.png"><IMG SRC="Graffiti_l.png n > 

</DTV> 

</body> 

HIGHLIGHTER 

FIG. 18 illustrates the steps for annotating a web page using the Highlighter. As described 
above, the user selects the Highlighter tool from the menu bar or by clicking on the icon 58, and selects 
the color for the highlighted text using the palette 78. These operations are performed by the browser 
control 106a under control of the module 134 of the main program 114. 

In a step 200 in FIG. 18, the program 114 determines that the Highlighter tool has been 
activated and gets the text to be highlighted from the browser control 106a in a step 202. The program 
1 14 then changes the color attribute of the highlighted text in the HTML document to the color selected 
from the palette 78 in a step 204. The final step 206 is to save the modified HTML document to the 
temporary directory. 

CYBER NOTES 

FIG. 19 illustrates the steps for annotating a web page by pasting a Cyber Note thereon. As 
described above, the user selects the Cyber Notes tool from the menu bar or by clicking on the icon 60, 
and selects a Cyber Note using the palette 82. These operations are performed by the browser control 
106a under control of the module 134 of the main program 114. 

In step 210 in FIG. 19, the program 1 14 determines that the Cyber Notes tool has been activated 



- 14 - 



WO 00/07110 



PCT/US99/17432 



and gets the mouse position from the browser control 106a in a step 212. Since Internet Explorer does 
not support drag and drop, this runctionality is emulated by the present module 138. 

More specifically, the module 138 uses the browser control 106a to generate an outline display 
or rectangle at the position of the cursor in a step 214. Preferably, the upper left corner of the rectangle 
is at the cursor position. Then, the module 138 checks to see if the left mouse button was clicked in 
a step 216, indicating that the position for the Cyber Note has been decided by the user. 

If yes, the module 138 inserts the Cyber Note at the cursor position in a step 218. If not, the 
module 138 loops back to the step 212 to get the new cursor position and redisplay the rectangle at the 
new position. 

The next step 220 is to utilize the browser control 106a to display a text box or dialog box to 
enable the user to enter text into the Cyber Note using the keyboard. A step 222 determines if a left 
mouse key click has been generated, indicating completion of the text insertion operation. If yes, the 
HTML file is modified and saved in the temporary directory, and the Cyber Note is saved as a PNG file. 
If no, the module 138 loops back to step 220 to input more text. 

The program 114 modifies the HTML document by inserting a CSS element into the above 
described table which references the stored Cyber Note PNG file. The CSS element specifies the 
position of the Cyber Note and the Z-order of the file. The format specified by Microsoft for a file 
named H Cyber_Note_l.png" is: 

<DIV dataFormatAs=TEXT id="Note:LM Cyber_Note_l.png" 

style="COLOR: #000000; CURSOR: auto; HEIGHT: 114px; 

LEFT: 202px; OVERFLOW: hidden; POSITION: absolute; 

TOP: 358px; VISIBILITY: visible; WIDTH: 189px; 

Z-INDEX: 258"><FONT face= H Courier New" size=2 

style= M CURSOR: auto; TEXT-DECORATION: none">Cyber Note 

l</FONT> </DIV> 

Again, this format may not display properly with Netscape Navigator. A format which will 

work is: 

<style type- 'text/css M > 

LM_Cyber_Note_l.png { color: #000000; height: 114px; 
width: 189px; left: 202px; top: 358px; overflow: 
hidden; position: absolute; visibility: visible; 
z-index: 258; } 

</style> 
<body> 

<DIV ID= M LM_Cyber_Note_l.png"><IMG SRC= n LM 

Cyber_Note_l.png<FONT face=" Courier New" size=2>Cyber 
Note l</FONT> 

</DIV> 
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</body> 

DIGITAL STICKERS 

An annotation operation for pasting a Digital Sticker on a web page is essentially similar to that 
for a Cyber Note as described above, except that a Digital Sticker does not have a text box for inserting 
a message. The operations for inputting text are therefore omitted. 

Digital Stickers are saved as PNG files. A CSS element referencing the PNG file is inserted 
in the table in the HTML document with the position of the Digital Sticker and the Z-order for the file 
specified. The format specified by Microsoft for a Digital Sticker file named "Digital JSucker_l.png'' 
is: 

<P style="CURSOR: auto"><IMG id=Sticker 

src= ,, Digital_Sticker^l.png n style="CURSOR: auto; 

LEFT: 225px; POSITION: absolute; TOP: 232px; 

VISIBILITY: visible; Z-INDEX: 256">   </P> 
A CSS format for a Digital Sticker which will display properly in Netscape is: 

<style type= ,, textycss"> 

Digital„Sticker_l.png { position: absolute; top: 
232px; left: 225px; visibility: visible; 
z-index: 256 } 

</style> 
<body> 

<DIV ID= n Digital_Sticker_Lpng H > 
<IMG SRC="Digital^Sticker_l.png ,, x/DIV> </body> 

SCRAPBOOK 

The scrapbook module 142 uses the browser control 106a and engine 106b to get the elements 
from an HTML document which correspond to blocked text on the displayed web page. The operation 
is similar to conventional cut and paste except that the content is pasted into a scrapbook HTML 
document. 

A scrapbook is created to encapsulate the pasted HTML content in a fixed width table in the 
same manner as described above for the preprocessor 112. Preferably, each block of HTML content is 
inserted into the scrapbook HTML document in a separate subtable. This prevents any attributes for one 
block of content from affecting another block. 

E-MAIL 

The operation of the E-mail module 1 1 6 is illustrated in FIG. 20. As described above, the user 
selects the E-mail function from the menu bar or by clicking on the icon 66. 

The first step is to set up the HOTW program 102 as an E-mail client. This causes a message 
composition form (not shown) to pop up from the user's default E-mail program, into which the user 
inputs the E-mail address of the addressee, an optional message and any other desired information. 

In the next step 232, the HTML document and any ancillary files are compressed into a self- 
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extracting cabinet file. This is "CAB" file with an extension of ".exe", and does not require a separate 
decompression program for extraction. The cabinet file is then transmitted to the addressee as an 

attachment to the message. 

If the recipient of the E-mail has HOTW, the attachment will be opened as a HOTW file. If 
not, the attachment will be opened in the recipient's default web browser. 

PRINTING 

The printing functions provided by the MSHTML.DLL file which constitutes the parsing and 
rendering engine 106b are limited. MSHTML.DLL is unable to do print preview. Also, if a document 
is too wide, part of the document is cut off. The present printing module 118 overcomes these 
deficiencies. 

The printing operation is illustrated in FIG. 21. As described above, toe user selects the print 
function from the menu bar or by clicking on the icon 44. The first step 240 of the process is to draw 
the content of the HTML document to a virtual screen, such as a block of memory locations in the RAM 
12. Then, the individual actions are parsed, broken down and rendered into pages which fit a printer 
in a step 242. Preferably, each page will have a fixed size such as 8 1/2 x 11 inches. Finally, the 
individual pages are rendered to the printer in a step 244. 

SERVER IMPLEMENTATION 
FIGs. 22 to 25 illustrate alternative embodiments of the present invention in which the storage 
of documents and/or execution of the annotation program are performed on a server under control of a 
user at a local terminal which is connected to the server by the internet or other network (Local Area 
Network (LAN), Wide Area Network (WAN), etc.). 

FIG. 22 illustrates a system 300 in which anser terminal 302 is connected to a server 304 over 
the internet 306. The user terminal 302 can be a personal computer including the conventional elements 
illustrated in FIG. 1. Alternatively, the terminal 302 can be a "dumb" terminal, including a display unit 
308, an input device 310, and an elementary processor and small amount of memory for running a low 

level control program (not shown). 

The user terminal 302 can also have a configuration other than a conventional desktop or laptop 
personal computer. For example, the present invention can be applied to a set-top unit such as used to 
connect a user to the internet using a conventional television receiver. In this case, the display unit 308 
is the television receiver and the input unit is a remote control unit for the set-top unit. Another 
applicable type of user terminal 302 is a small hand-held unit. 

The server 304 is generally larger than a personal computer including, for example, a mainframe 
computer. The server 304 includes a storage 312 which typically includes a bank of hard drives. An 
annotation program 314 according to the present invention is stored in the storage 312 and executed by 
a processor 316. 

The storage 312 stores at least one hypertext document 318, preferably a number of linked 
documents and ancillary image and other files, including original documents for normal browsing and 
modified documents including user annotations. The annotations files 320 themselves are stored in the 
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storage 312 in the same folder with or in a different folder from the modified documents. 

The user terminal 302 may or may not run a browser program. In the latter case, the 
functionality of the browser is provided on the server 304. A user utilizes the input unit 310 of the user 
terminal 302 to browse the web site on the server 304 and cause a selected web page corresponding to 
a document 318 to be displayed on the display unit 308. The program 324 enables the user to annotate, 
save, print and E-mail a web page in the manner described above. 

The main difference is that the program 314 is stored and executed on the server 304 instead 
of on the user terminal 302. Also, annotated hypertext documents, ancillary files and annotation files 
are stored on the server 304. This configuration enables a user to access the internet using a very simple 
and inexpensive user terminal 302 with a minimal or no file storage capability. A variation of this 
arrangement is that a modified version of an annotated hypertext document need not be stored on the 
server 304. A modified version can be created and the annotations 320 applied thereto "on-the-fly" in 
response to a user accessing the hypertext document at a subsequent time. 

FIG. 23 illustrates another server implemented embodiment of the invention in which like 
elements are designated by the same reference numerals used in FIG. 22. In a system 322, a user 
terminal 324 is provided with a small amount of storage which is sufficient to store the annotation files 
320. The hypertext document files 318 are stored on the server 304. The program, here designated as 
326, is modified to download the annotation files 320 to the user terminal 324. 

FIG. 24 illustrates another system 328 which is similar to the system 322 of FIG. 23 except that 
both an altered hypertext document 330 and associated ancillary and annotation files 320 are downloaded 
to a user terminal 332 by a program 334 after the annotation operation has been performed on the server 



304. 



FIG. 25 illustrates another server implemented system 336 which is similar to the stand-alone 
embodiment described above with reference to FIGs. 1 to 21 except that the documents 318 are stored 
on the server 304. A user terminal 337 includes the processor 326 which executes an annotation 
program 338 stored in the storage 312 which is located in the user terminal 337. The server 318 can 
store modified documents as well as original documents 318, or modified documents can be created on- 
the-fly as described above. 

In the embodiments described above, the present annotation program has been described as a 
conventional program which is executed directly by a processor. However, the present invention is not 
so limited. It is within the scope of the invention to provide the annotation functionality as a Java 
applet, ActiveX control, or XML functionality, which is stored on a server and downloaded to a user 
terminal together with a hypertext document. The applet or control is launched by a browser or other 
application program and executed using the user terminal's processor. 
TNDUSTRIAT. APPLICABILITY 

MODIFICATIONS 

Various modifications will become possible for those skilled in the art after receiving the 
teachings of the present disclosure without departing from the scope thereof. The following 
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modifications are exemplary only, and are not intended to be inclusive or in any way imitating the scope 
of the invention. 

1. The present invention has been described above as being used in conjunction with the 
Microsoft Internet Explorer 4.0 web browser. However, the invention is not so limited, and the present 
system can be modified to work in conjunction with Netscape Navigator or any other commercially 
available or proprietary web browser. The present program can also be provided as a module (e.g. plug- 
in) of a web browser or other application program such as a word processor. 

2. It is also within the scope of the invention to modify the system to function in a stand 
alone manner without an external web browser. This can be accomplished using non-MSHTML. DLL 
dependant software with comparable functionality, in combination with software which provides the 
functionality of the browser control 106a and associated elements. 

3 Although the invention has been described above as operating on hypertext documents 

written in the HTML language, the scope of the invention is not so limited. The present invention is 
applicable to any type of hypertext document which can include a link to another document. 

Other hypertext document formats which are compatible with the invention include Standardized 
Generalized Markup Language (SGML), Dynamic Hypertext Markup Language (DHTML) and 
Extensible Markup Language (XML). These formats are listed as examples only, and are not limitative 
of the scope of the invention. For example, the invention can be applied to a hypertext format in which 
the text elements are binary encoded. 

4. As another typical modification within the scope of the invention, it is possible to use 
another file format. The PNG format is preferred because it is possible to easily display a PNG file with 
a transparent background in Internet Explorer. This enables the PNG annotation files to be overlaid on 
the main web page content without the background areas of the PNG image obscuring underlying main 
content. 

However, Netscape Navigator does not support transparent PNG files. For this reason, the 
transparent GIF file format can be substituted for the PNG format. Transparent GIF files are supported 
in both of these web browsers. Other suitable file formats are further contemplated within the scope of 
the invention. 

5. Alternative methods of positioning the Graffiti pen, Digital Sticker and Cyber Note 
graphic images on a page are possible within the scope of the invention. The method which is described 
above and implemented in the HOTW product is to enclose the page in a fixed width table in which a 
graphic image is referenced by an element in a cell of the table. The position of the cell determines the 
position of the image on the page. 

It is further possible to define a position for a graphic image by overlaying the page with a 
virtual grid, and defining the image position as a pair of coordinates on the grid. Yet another alternative 
is to specify the image position as a offset from a predetermined position or anchor on the page, for 
example from the upper left comer of the page. 

6. The embodiment of the invention described above preprocesses a hypertext document 
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perform corresponding operations to consolidate annotation after the annotation has been completed. 
This arrangement can be described as "postprocessing". 

In the postprocessing embodiment, the web page is displayed on the display unit, and the 
annotation images are created and displayed as overlays on the web page. The page is not modified at 
this time to incorporate the annotations. After annotation is completed, the page is modified and saved. 

The modifications which are performed prior to saving preferably include enclosing the page 
content in a fixed width table as described above with the font attributes fixed. Cascading style sheet 
elements pointing to the annotation files are also inserted as described ahove. Active and external links 
can be de-activated or left active. The links to the ancillary files can be changed to reference the local 
copies as part of the postprocessing operation. 

It is also within the scope of the invention to combine preprocessing and postprocessing. For 
example, the links to the ancillary files can be changed during preprocessing, and the other modifications 

performed during postprocessing. 

7. A primary application of the present invention is to capture, annotate, save and transmit 
web pages downloaded from the internet or other network. However, the web pages which are 
processed using the invention can be produced locally using a text editor to create a text document 
consisting of HTML elements and tags. Web pages can also be obtained from a friend or business 
associate on a floppy disk or other portable storage element and loaded into a user's personal computer 

which runs the program. 

8. It is also within the scope of the invention to annotate graphic images which may be 
in a propriety file format and which are not desired to be altered. This modification is accomplished 
by painting a graphic image on the display unit, and using the present program to create a hypertext 
document which is overlaid on the graphic image. 

A hypertext document overlay which does not include any display elements (text or images) 
will be transparent. However, using the above described annotation tools to create Graffiti lines, Digital 
Stickers and Cyher Notes will cause CSS elements to be inserted into the hypertext document and cause 
the corresponding annotation elements to be superimposed on the graphic image. This method can be 
used for any suitable type of graphic image file, such as Autodesk Autocad* files ( dwg), Adobe Portable 
Document Format (PDF) files (.pdf), etc. 
njTWTCTPTAT. APPIJCABILITY 

The present invention generally relates to the art of electronic communications, and more 
specifically to a system for capturing, annotating and transmitting images of internet World Wide Web 
(WWW) pages. 
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WE CLAIM 

METHOD CLAIMS 
Generic 

1. A programmed digital computer system (10), comprising: 

a display unit (16) for displaying a hypertext document (154); 
an input unit (22); 
a processor (14); and 

a storage (20) for storing a program (102) which is executed by the processor (14), the 
program (102) including an annotator (114) for annotating the page (38) by modifying the document 
(154) in response to input from the input unit (22), said annotation including painting an image 
(150),(152),(154) on the page (38) at a position designated by the input unit (22). 

LOCATION OF PROGRAM AND FILES 

2. A system (10) as in claim 1, in which: 

the system (10) comprises a user terminal (10) which is connected to a network (26) 

having accessible hypertext documents; 

the user terminal (10) comprises the display unit (16), input unit (22), processor (14) 

and storage (20); and 

the program (154) comprises instructions for downloading a hypertext document (154) 
from the network (26). 

3. A system (10) as in claim 1, in which: 

the system (10) comprises a user terminal (10) which is connected to a network (26) 

having accessible hypertext documents; 

the user terminal (10) comprises the display unit (16), input unit (22), processor (14) 

and storage (20); 

the system (10) further comprises a browser (106) for browsing and downloading 
hypertext documents from the network (26); and 

the program (102) is part of the browser (106). 

4. A system (10) as in claim 3, in which the program (102) is a plug-in for the browser 

(106). 

5. A system (10) as in claim 1, in which: 

the system (10) further comprises an application program (26); and 
said program (102) is part of the application program (106). 
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6. A system (10) as in claim 1, in which: 

the system (10) comprises a user terminal (10) and a server which are interconnected 

by a network (26); 

the user terminal (10) comprises the display unit (16) and the input unit (22); and 
the server comprises the processor (14) and the storage (20). 

7. A system (10) as in claim 1, in which: 

the system (10) comprises a user terminal (10) and a server which are interconnected 

by a network (26); 

the user terminal (10) comprises the display unit (16), input unit (22) and processor 

(14); and 

the server comprises the storage (20). 

8. A system (10) as in claim 1, in which the page (38) is an overlay for a separately 
created graphic image file. 

9. A system (10) as in claim 8, in which: 
the image is substantially opaque; and 

areas of the page (38) other than the image (150),(152),(154) are substantially 

transparent. 

10. A system (10) as in claim 1, in which: 

the system (10) comprises a user terminal (10) and a server which are interconnected 

by a network (26); 

the user terminal (10) comprises the display unit (16) and the input unit (22); 

the document (154) is stored on the server; and 

the program (102) modifies the document (154) on the server. 

11. A system (10) as in claim 10, in which: 

the server comprises the processor (14) and the storage (20); and 
the processor (14) executes the program (102) on the server. 

12. A system (10) as in claim 10, in which: 

the image (150),(152),(154) is represented by an image file (150),(152),(154) which is 

stored in the storage (20); and 

the user terminal (10) comprises the storage (20). 
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POSITIONAL REFERENCE 

13. A system (10) as in claim 1, in which the program (102) is configured to modify the 
document (154) to establish a positional reference for the image (150),(152),(154). 

14. A system (10) as in claim 13, in which the positional reference comprises a virtual grid 
which is overlaid on the document (154). 

15. A system (10) as in claim 13, in which the positional reference comprises a table in 
which the document (154) is enclosed. 

16. A system (100 as in claim 15, in which the table comprises a plurality of cells. 

17. A system (10) as in claim 15, in which the table has a fixed width. 

18. A system (10) as in claim 13, in which the positional reference comprises an anchor 
point in the document (154). 

19. A system (10) as in claim 18, in which the program (102) is configured to calculate 
the position designated by the input unit (22) as an offset from the anchor point. 

20. A system (10) as in claim 19, in which the program (102) is configured to modify the 
document (154) so as to have a fixed width. 

DIGITAL STICKERS AND CYBER NOTES 

21. (10) A system as in claim 1, in which: 

the image (150),(152),(154) is represented by an image file (150),(152),(154); and 
the annotator (114) is configured to modify the document (154) to include an element 
which references the image file (150),(152),(154). 

22. A system (10) as in claim 21, in which the element comprises a Cascading Style Sheet 

(CSS). 

23. A system (10) as in claim 22, in which the CSS specifies the position for the image 
(150),(152),(154) and a Z-order for the image file (150),(152),(154). 

24. A system (10) as in claim 21, in which the image file (150),(152),(154) comprises a 
Portable Network Graphic (PNG) file. 
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25. A system (10) as in claim 21, in which the annotator (114) is configured to cause an 
outline representing the image (150),(152),(154) to be displayed on the page (38) at a position designated 
by the input unit (22), and to cause the image (150),(152)>(154) to be painted onto the page (38) at the 
position of the outline in response to a subsequent input from the input unit (22). 

26. A system (10) as in claim 21, in which the annotator (114) is configured to enable a 
user to select an image (150),(152),(154) from a set of available images(150),(152),(154). 

27. A system (10) as in claim 21, in which the annotator (114) is configured to cause a 
dialog box (82) containing a set of available images (150),(152),(154) to be displayed which enables a 
user to select an image from the set. 

28. A system (10) as in claim 21, in which the annotator (114) is further configured to 
enable a user to create an image (150),(152),(154) by importing an image file (150),(152),(154). 

29. A system (10) as in claim 21, in which the annotator (114) is further configured to 
enable a user to input text into a box in the image (150),(152),(154) using the input unit (22). 

30. A system (10) as in claim 29, in which the annotator (1 14) is configured to cause an 
outline (84) representing the image (150),(152),(154) to be displayed on the page (38) at a position 
designated by the input unit (22), cause the image (150),(152),(154) to be printed onto the page (38) at 
the position of the outline (84) in response to a subsequent input from the input unit (22), and 
subsequently enable a user to type text into a box in the image (150),(152),(154). 

31. A system (10) as in claim 29, in which the annotator (114) is further configured to 
enable a user to select an attribute for the text. 

32. A system (10) as in claim 29, in which the annotator (114) is further configured to 
cause a dialog box (82) to be displayed which enables a user to select an attribute for the text. 

GRAFFITI PEN 

33. A system (10) as in claim 1, in which the annotator (1 14) is configured to enable a user 
to produce the image (150),(152),(154) by drawing a line on the page (38) using the input unit (22). 

34. A system (10) as in claim 33, in which the annotator (1 14) is configured to sense cursor 
positions designated by the input unit (22) to represent the line, store a representation of the cursor 
positions in a position file, and modify the document (38) to include an element which references the 
position file. 
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35. A system (10) as in claim 34, in which the element comprises a Cascading Style Sheet 

(CSS). 

36. A system (10) as in claim 35, in which the CSS specifies a position for the image 
(150),(152),(154) and a Z-order for the position file. 

37. A system (10) as in claim 34, in which the position file comprises a Portable Network 
Graphic (PNG) file. 

38. A system (10) as in claim 33, in which the annotator (114) is configured to enable a 
user to select a color for the line. 

39. A system (10) as in claim 33, in which the annotator (114) is configured to cause a 
color pick palette (78) to be displayed which enables a user to select a color for the line. 

HIGHLIGHTER 

40. A system (10) as in claim 1, in which the annotator (1 14) is configured to enable a user 
to highlight selected text on the page (38). 

41. A system (10) as in claim 40, in which the annotator (1 14) is configured to modify the 
document (154) by changing a color attribute of highlighted text to a selected highlight color. 

42. A system (10) as in claim 40, in which the annotator (114) is configured to enable a 
user to select a color for the text. 

43. A system (10) as in claim 40, in which the annotator (114) is configured to cause a 
color pick palette (78) to be displayed which enables a user to select a color for the text. 

SCRAPBOOK 

44. A system (10) as in claim 1, in which the annotator (1 14) is configured to enable a user 
to create a scrapbook file (142), and insert a selected section of a page into the scrapbook file (142) 
using the input unit (102). 

45. A system (10) as in claim 44, in which the annotator (114) is configured to enable a 
user to block and copy a selected section and paste the selected section into the scrapbook file (142). 

E-MAIL 

46. A system (10) as in claim 1, in which the program (102) further comprises an electronic 
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mail module (116) configured to transmit the document (154) as an electronic mail message. 

47. A system (10) as in claim 46, in which the electronic mail module (116) is configured 
to compress and transmit the document (154) as a self-extracting file. 

48. A system (10) as in claim 46, in which: 

a local copy of an ancillary file which is referenced by a hypertext link in the document 

(154) is stored in the storage (20); and 

the electronic mail module is further configured to transmit the local copy together with 

the document (154). 

49. A system (10) as in claim 48, in which the electronic mail module (116) is configured 
to compress and transmit the document (154) and the local copy as a single self-extracting file. 

PRINTING 

50. A system (10) as in claim 1, in which the program (102) further comprises a printing 
module (118) configured to print the page (38). 

51. A system (10) as in claim 50, in which the printing module (1 16) is configured to draw 
the page (38) to a virtual screen to capture actions performed and thereby render content, parse and 
break down the actions and render the actions into pages which fit in a designated printer, and render 
the pages to the printer. 

PREPROCESSOR FUNCTIONS 

52. A system (10) as in claim 1, in which the program further comprises a preprocessor 
(112) for preprocessing the document (154) into a predetermined format which facilitates annotation. 

53. A system (10) as in claim 52, in which the preprocessor (112) is configured to 
determine if a hypertext link which references an ancillary file of a predetermined type is present in the 
document (154) and, if a link is present, store a local copy of the ancillary file in the storage (20) and 
modify the document (154) by changing the link to reference the local copy. 

54. A system (10) as in claim 52, in which the predetermined type of ancillary file is an 
image file. 

55. A system (10) as in claim 52, in which the (1 12) preprocessor is further configured to 
modify the document (154) such that the page (38) has a predetermined, fixed width. 
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56. A system (10) as in claim 52, in which the preprocessor (1 12) is configured to modify 
the document (154) such that the page (38) is enclosed in a table having a predetermined, fixed width. 

57. A system (10) as in claim 52, in which the preprocessor (112) is further configured to 
modify the document (154) to disable word wrapping. 

58. A system (10) as in claim 52, in which the preprocessor (112) is further configured to 
determine if an external hypertext link of a predetermined type is present in the document (154) and, 
if present, deactivate the external hypertext link. 

59. A system (10) as in claim 52, in which the preprocessor (112) is further configured to 
determine if an active hypertext link of a predetermined type is present in the document (154) and, if 
present, deactivate the active hypertext link. 

60. A system (10) as in claim 52, in which the preprocessor (112) is further configured to 
modify the document (154) by fixing a default font setting. 

POSTPROCESSOR FUNCTIONS 

61. A system (10) as in claim 1, in which the program (102) further comprises a 
postprocessor (112) for postprocessing the document (154) into a predetermined format which 
consolidates annotation. 

62. A system (10) as in claim 61, in which the postprocessor (112) is configured to 
determine if a hypertext link which references an ancillary file of a predetermined type is present in the 
document (154) and, if a link is present, store a local copy of the ancillary file in the storage (20) and 
modify the document (154) by changing the link to reference the local copy. 

63. A system (10) as in claim 61, in which the predetermined type of ancillary file is an 
image file. 

64. A system (10) as in claim 61, in which the postprocessor (112) is further configured 
to modify the document (154) such that the page (38) has a predetermined, fixed width. 

65. A system (10) as in claim 61, in which the postprocessor (112) is configured to modify 
the document (154) such that the page (38) is enclosed in a table having a predetermined, fixed width. 

66. A system (10) as in claim 65, in which the postprocessor (112) is further configured 
to modify the document (154) to disable word wrapping. 
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67. A system (10) as in claim 61, in which, the postprocessor (112) is further configured 
to modify the document (154) by fixing a default font setting. 

JAVA APPLET, ActiveX CONTROL 

68. A system (10) as in claim 1, in which at least part of the program (102) is downloaded 
from a server together with the document (154). 

69. A system (10) as in claim 68, in which the program (102) comprises an applet. 

70. A system (10) as in claim 68, in which the program (102) comprises a control. 
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